//================================================================================================================================================= // Dev Log of [Camera Take Detection] // // by Guanghan Ning [gnxr9@mail.missouri.edu] // // Update since stable version of Chen Huang's work. //================================================================================================================================================= During Christmas Vacation: 1. Choosing the least-motion frame as the key frame of a shot boundary, rather than the default last frame. (1). It improves shot boundary quality (2). And better camera take representation (3). Better output frame quality 2. Cut off the subtitles before the raw pixel subtraction at the shot boundary detection stage. 3. Fixed bug: originally we "can" start from any frame, but it did not really work. Now we can start from any frame. 4. Fixed bug: Originally the program consumes too much memory and it crashes, now the garbage is collected in time. Memory usage is now very stable. --------------------------------------------------------------------------------------------------------------------------------------------------- During CES: 1.5~1.7.2015 1. Combine the motionless frame selection and original frame selection: When the motion of a shot boundary is not significant, use the last frame as the key frame, because content-wise, the composition of the picture is usually good in the last frame of a shot. 2. Choose the key frame of a take to output, instead of the first shot boundary's middle frame, which is too arbitrary. The output frame will be less blurred. 2. Choose the least-motion key frame from all the shot boundaries of one take, as the key frame of the take. The intuitive is that the best frame of the first/middle/last shot boundary may not be the best; there may be other shots whose key frame is more appropriate to be the key frame of the take. Before this update, some key frames of shots are still with some motion. --------------------------------------------------------------------------------------------------------------------------------------------------- Since 1.8.2015 1. Add one more step of resize. (1). Ensure the key points extracted are not changed, so that the camera take quality is not reduced. (2). Decrease the running time of the program. 2. Skip a frame before processing the next frame. (1). Shorten the processing time by almost a half. --------------------------------------------------------------------------------------------------------------------------------------------------- Since 1.12.2015 1. Fixed a bug: There was a bug in [Skipping frame to speed up the processing time]. The result gets worse than the original method, with more output frames, some of which are also repetitive. Now skipping one frame each time, with the same result as not skipping frames. 2. Updated Json file: (1). added time stamps in addition to pure frame number, since videos may have different fps; (2). added key frame based on motion, replacing the original arbitary middle frame; added its corresponding time stamp. 3. Introduced an image buffer to replace vc.set(), which will save a lot of computation. After bug fixing, it now outputs exactly the same result with less time. ---------------------------------------------------------------------------------------------------------------------------------------------------- Since 1.13.2015 1. Skipping frame is improved: We can skip for every 2*N frames. N is an integer. 2. Resizing sololy the images for motion estimation, not changing the size for the images that conduct raw pixel difference. The speed is much faster. With all speedup techniques applied, the test time drops from 380000 to 63000. It drops to 47000 if skipping 4 frames instead of 2. It takes around 30 minutes to complete [duiwoeryankeaideta03.flv] 3. change cropping from 7/8 to 6/8. 4. Performance improved! Less blurred images and less un-merged takes(that should merge). (1). Do not output those takes whose key frames have huge motion points. (2). Introduced new thresholds: private static final double MV_RESIZE_RATE= 0.5; private static final double MV_CROP_RATE= 0.75; private static final double MV_TH= 0.6; // skip camera take output private static final int MIN_KP2 = 30; //50; // skip camera take merging private static final int MIN_KP = 30; //110; // skip camera take output (Lower MIN_KP2 and MIN_KP2 to allow more merging) ------------------------------------------------------------------------------------------------------------------------------------------------------ Since 1.15.2015 1. Update json structure. Added a flag for each shot boundary in a take, indicating which shot boundary has the key frame for the take. ------------------------------------------------------------------------------------------------------------------------------------------------------ Since 1.16.2015 1. Fixed a bug! Chen's original code has a hidden bug. stFrame is not updated when a shot is skipped with too little frames. It causes wrong key frame selection in later processing. This is why it is alright for a short frame range but wrong for a big range! Because for a big range, there will exist such shots whose frames are too little and skipped and encounter the hidden bug. Improvement: Repetitive images are much less! ------------------------------------------------------------------------------------------------------------------------------------------------------ Since 1.19.2015 1. Save key points and keyFrame image into new structures, for better storage and access. 2. Add Restriction: For key Frames that are too close, skip the latter. This improves the output quality. ------------------------------------------------------------------------------------------------------------------------------------------------------ Since 1.20.2015 1. Fixed a bug: When there is no camera take output, use the whole clip as the take. 2. Added a flag to symbolize which shot boundary contains the take's key frame. ------------------------------------------------------------------------------------------------------------------------------------------------------ Since 4.20.2015 1. Updated the Color histogram comparison method Improved the performance for small resolution videos. The number of output frames improved. 2. Add the [throwBlur] property to each camera take so that programs that read in this json file will know which take should be thrown ------------------------------------------------------------------------------------------------------------------------------------------------------- Since 8/6/2015 Delete camera takes that does not meet the following criterions: 1. # of key points okay 2. motion vector okay 3. Overall take duration okay ------------------------------------------------------------------------------------------------------------------------------------------------------- Since 8/10/2015 Add additional processing after camera take generation: 1. Merge those shots within the same take, under the condition that these shots are 0.25s or closer to each other. [This is to suppress long take spliting into multiple shots] -------------------------------------------------------------------------------------------------------------------------------------------------------- Since 8/24/2015 Fixed issue: Take id number not sequntial due to camera take deletion with criterion the id of the output camera take is now sequence number